Users' performance in lab and non-lab environments through online usability testing:A case of evaluating the usability of digital academic libraries' websites

The factors related to the environment in which users operate may be of a vital importance when trying to understand how they experience a particular system. It is required that we find out how we can get to know those factors to investigate if they affect the users' performance in usability testing. An online usability study has emerged that can be attempted by a large, varied pool of users' anywhere with an Internet connection. Would the usage of an online usability study help to give comprehensive insight and an understanding of the whole user experience? That is especially interesting if the user operates remotely, as we are unaware of what the users might experience while performing the test (e.g., distractions and type of device used to attempt the test). Accordingly, a pilot study of ongoing research was conducted. An identical online usability-testing tool (Loop 11) through which to apply the online usability study was used in two environments: unrestricted (the user's natural environment), and totally restricted (a simulated lab environment). Ten subjects completed the test in the restricted environment and 20 completed it in the unrestricted environment. All of the subjects were asked to perform predefined search tasks on digital libraries' websites. Their performance was analyzed and compared against the two different environments. The results showed that online usability testing is a feasible method to gain comprehensive insight into how users attempt usability testing in a non-lab environment. The results of whether different environments affect test performance show no valuable differences in most of the study's measurements. The test subjects were frequently multitask while they performed the usability testing in an unrestricted environment, but they were highly distracted if they personally interrupted. The results encourage the researcher to conduct a formal version of this study to further examine the learned lessons from the pilot study

To intervene or not to intervene: An investigation of three think-aloud protocols in usability testing

This paper presents the results of a study investigating the use of three think-aloud methods in website usability testing: the concurrent think-aloud, the speech-communication, and the active intervention methods. These three methods were compared through an evaluation of a library website, which involved four points of comparison: overall task performance, test participants’ experiences, the quantity and quality of usability problems discovered, and the cost of employing the methods. Data were collected from 60 individuals, with 20 participants allocated to each testing method, who were asked to complete a set of nine experimental tasks. The results of the study revealed that the three variations enabled the identification of a similar number of usability problems and types. However, the active intervention method was found to cause some reactivity, modifying participants’ interaction with the interface, and negatively affecting their feelings towards the evaluator. The active intervention method also required much greater investment than did the other two methods in terms of evaluators' time

Eye tracking in retrospective think-aloud usability testing: Is there added value?

Eye tracking is the process of recording users’ eye movements while they are looking at the location of an object. In usability testing, this technique is commonly used in combination with think-aloud protocols. This paper presents an experimental study involving 24 participants; with the aim of comparing two variants of retrospective think-aloud (RTA) methods, that is, video-cued RTA method and gaze-cued RTA method, to address the value of having an extra eye-cue in retrospective think-aloud usability testing. Results suggest that both RTA variants are effective in detecting major usability problems. Moreover, the combination of eye tracking techniques and think-aloud protocols can further help evaluators to detect more usability problems, especially minor navigational and comprehension problems. It also helps participants to remember their behavior details, such as what they were looking at on a web page, as mouse movement alone might not be representative of their actual thoughts. Nevertheless, we found that participants might become distracted while seeing their eye movement, which can affect their verbalization performance and, hence, they might experience longer silence periods

Current Issues of Usability Testing

System usability can be measured through various methods. One of the more important and widely ­employed techniques is 'usability testing', where asks, number of users, evaluators, and other factors are the main ­elements. This paper reviews usability testing together with current issues that can influence usability testing results, both negatively and positively. It also reviews web usability testing. In addition, in this paper, usability testing in the future is considered in order that improvements may be made

Introducing IT Purchased from Third Party Vendors

This paper highlights the key areas of a case study which reviews the implementation of a packaged Information Systems solution purchased from a third party vendor, into one of the largest insurance companies in the UK. A number of perspectives are shown from various levels of staff ranging from the project team to the end-users in order to portray a balanced view of the implementation. Focus is placed on how these differing members of staff perceived the level of user involvement in order to explore whether critical success factors for a successful implementation differ between stakeholders